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1.  Overview.  This  guide  is  divided  Into  three  parts. 

T.  The  first  part  provides  the  evaluator  with  a  generalized  introduction 
to  the  evaluation  concept  including  a  basic  understanding  of  the  evaluation 
procedures  and  background  information.  Procedures  for  recommending  changes  in 
the  evaluation  process  are  also  included. 

b.  The  second  part,  attachment  1,  contains  a  one-page  explanation  of  each 
question  used  In  the  evaluation.  Applicable  examples  and  any  unique  definitions 
are  provided  for  each  question.  The  questions  are  arranged  into  six  groups 
separated  by  pages  that  define  each  group. 

c.  The  third  part,  attachment  2,  contains  a  cross-reference  Index  of 
questions  versus  subject  matter.. 

\ 


Background. 


a.  In  the  past,  the  operator-machine  Interface  for  a  piece  of  computer- 
driven  equipment  has  been  evaluated  on  an  exception-only  basis;  l.e.,  each 
operator  would  address  only  those  areas  of  the  Interface  with  which  he  was 
particularly  disturbed.  This  method  of  analysis  naturally  resulted  In  highly 
subjective,  nonspecific  results.  Operators  would  rate  the  Interface  "good"  or 
"bad"  according  to  the  number  and  difficulty  of  problems  each  encountered. 
Experienced  operators  often  have  less  problems  than  Inexperienced  operators 
merely  because  they  have  learned  to  live  with  system  peculiarities. 

b.  Highly  subjective  evaluations  are  undesirable  because  they  may  not 
yield  accurate  estimates  of  operational  capabilities  and  they  may  not  yield 
descriptions  of  specific  problems  that  need  to  be  fixed  to  Increase  operational 
capabilities. 

c.  HQ  AFTEC,  in  an  effort  to  decrease  the  subjectivity  of  analyses  of  the 
software  portion  of  the  operator-machine  Interface,  has  generated  the  Software 
Operator-Machine  Interface  Questionnaire  (SOMIQ).  The  concept  behind  the  SOMIQ 
Is  that  each  operator/evaluator  Is  directed  to  Isolate  and  consider  each  of  a 
number  of  quality  factors  about  the  equipment  being  evaluated.  Through  this 
organized  approach,  the  operators  all  consider  the  same  subjects,  thereby 
providing  a  more  uniform  analysis.  Furthermore,  the  operators  are  guided  to 
consider  subjects  which  they  might  overlook  If  asked  to  generate  lists  of 
problem  areas.  Additionally,  Information  Is  obtained  as  to  which  aspects  of  a 
system  contribute  positively  to  operational  capabilities. 

d.  The  SOMIQ  was  conceived  In  the  spring  of  1978  and  the  first  version 
was  published  In  the  winter  of  1979.  That  version  was  advertised  widely  through 
the  Air  Force  Systems  Command’s  and  Air  Force  Logistics  Command's  embedded 
computer  resource  newsletters  and  through  the  automatic  test  equipment  (ATE) 
newsletter  published  by  the  Navy.  Approximately  200  Interested  parties  re¬ 
sponded  to  the  advertisements,  and  about  half  that  number  provided  comments  to 
Headquarters  (HQ)  Air  Force  Test  and  Evaluation  Center  (AFTEC).  In  addition, 
the  SOMIQ  has  been  used  by  several  AFTEC  test  teams  on  a  variety  of  systems 
since  the  first  version  was  published. 


3.  Evaluation  Method. 


a.  The  method  for  evaluating  the  software  portions  of  the  operator- 
machine  Interface  Is  based  on  the  use  of  a  closed-form  questionnaire  with 


2 


AFTECP  800-2,  Vol  4  1  Nov  82 


optional  written  comments.  This  questionnaire  Is  designed  to  determine  the 
extent  of  the  presence  of  certain  desirable  attributes  In  a  given  system. 

b.  The  desirable  attributes  addressed  by  the  questionnaire  are  divided 
Into  six  groups  called  characteristics.  A  complete  understanding  of  the  defi¬ 
nitions  of  the  characteristics  Is  of  prime  importance  to  an  accurate  evaluation; 
thus,  the  evaluator  should  study  these  definitions  carefully. 

4.  Definitions. 

a.  Assurabllity.  A  computerized  system  contains  the  quality  of  assur- 
ablllty  to  the  extent  that  It  aids  the  operator  in  validating  data,  avoiding 
errors,  and  correcting  errors  once  made.  A  system  which  has  been  designed  to 
aid  the  operator  In  error  avoidance  may  or  may  not  achieve  this  goal.  A  system 
should  also  be  designed  so  that  errors  are  easy  to  correct,  and  above  all,  so 
that  errors  do  not  have  catastrophic  effects. 

b.  Controllability.  A  computerized  system  contains  the  quality  of 
controllability  to  the  extent  that  It  allows  the  operator  to  direct  the  opera¬ 
tions  of  the  machine.  The  operator  must  be  able  to  direct  or  control  the 
operation  of  the  machine  In  order  to  utilize  It  effectively  and  efficiently. 

c.  Workload  reasonability.  A  computerized  system  contains  the  quality  of 
workload  reasonability  to  the  extent  that  the  tasks  required  of  the  operator  are 
within  the  operator's  capability  and  require  the  operator  to  perform  a  useful, 
meaningful  role.  Optimum  design  of  a  system  which  Involves  an  operator  and  a 
computerized  machine  takes  advantage  of  the  best  capabilities  of  both:  the 
machine  to  perform  repetitive  tasks  rapidly  and  the  operator  to  make  command 
decisions  Involving  unusual  situations. 

d.  Descriptiveness.  A  computerized  system  contains  the  quality  of 
descriptiveness  to  the  extent  that  the  operator  has  available  adequate  explana¬ 
tions  of  every  function  the  operator  Is  required  to  perform  and  every  function 
the  machine  performs.  The  operator  need  not  be  Informed  In  detail  of  every  task 
the  machine  performs,  but  there  definitely  are  certain  things  the  operator  must 
know  to  fulfill  the  mission.  The  questionnaire  relies  upon  the  knowledge  of  the 
operator  to  define  what  It  Is  the  operator  needs  to  know. 

e.  Consistency.  A  computerized  system  contains  the  quality  of  consis¬ 
tency  to  the  extent  that  the  behavior  of  the  machine  and  documentation  corre¬ 
sponds  to  the  expectations  of  the  operator.  There  should  be  a  near  one-to-one 
correspondence  between  what  the  machine  does,  what  the  documentation  says  It 
will  do,  and  what  the  operator  has  been  trained  to  expect  the  machine  to  do. 

f.  Simplicity.  A  computerized  system  contains  the  quality  of  simplicity 
to  the  extent  that  Information  presented  to  the  operator  or  entered  by  the 
operator  is  grouped  Into  short,  readily  understandable  structures.  Complicated 
data  structures,  data  entry  formats,  or  operator  manuals  all  require  the  opera¬ 
tor  to  spend  more  time  In  developing  an  understanding  of  the  system. and  may  have 
a  tendency  to  confuse  the  operator  as  well. 

5.  What  to  Evaluate.  The  software  operator-machine  Interface  evaluation  may 
be  conducted  In  one  or  more  of  several  areas  within  an  overall  weapons  system 
operational  test  and  evaluation.  The  evaluation  for  any  given  system  may 
Involve  one  or  more  weapon  system  operators  as  well  as  operators  for  one  or  more 
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levels  of  computer-driven  support  equipment.  Each  of  these  different  types  of 
operator  functions  must  be  evaluated  separately  from  the  other  since  the  design 
of  the  operator-machine  Interface  Is  different  for  each  type  of  operator. 

6.  When  to  Evaluate.  It  Is  left  to  the  software  test  manager  and  deputy  for 
software  evaluation  to  determine  when  to  perform  this  evaluation.  However,  It 
Is  appropriate  to  present  some  general  guidelines  and  considerations. 

a.  One  of  the  assumptions  made  In  the  design  of  the  SOMIQ  Is  that  the 
operators  performing  the  evaluation  are  familiar  with  the  equipment  being 
evaluated.  Therefore,  this  evaluation  should  not  be  performed  before  the 
evaluators  are  familiar  with  the  equipment. 

b.  The  evaluation  could  be  done  once  toward  the  middle  of  the  test  period 
and  again  at  the  end  of  the  test  period  to  see  what  effect  increased  operator 
experience  has  and  to  see  the  Impact  of  contractor  updates. 

7.  Evaluators. 

a.  One  of  the  major  assumptions  of  this  evaluation  method  is  that  the 
evaluators  have  a  working  knowledge  of  the  system  from  the  perspective  of  the 
specific  operator  station  being  evaluated.  The  more  experience  an  operator  has, 
the  more  accurate  the  judgments  which  the  evaluator  must  make.  The  bottom  line 
Is  that  the  most  experienced  operators  available  should  perform  the  evaluation. 

b.  However,  If  the  number  of  highly  experienced  operators  Is  exceedingly 
small  but  an  adequate  number  of  evaluators  of  lesser  experience  are  available, 
then  all  those  operators  should  perform  the  evaluation.  HQ  AFTEC  can  run  the 
analysis  of  results  by  comparing  and  combining  the  results  from  both  groups  of 
evaluators. 

c.  Occasionally,  the  situation  will  arise  In  which  no  evaluators  with  any 
actual  operator  experience  exist.  This  might  occur  when  no  hardware  has  been 
produced  and  the  evaluation  Is  being  performed  on  a  prototype  system.  Such  a 
situation  will,  of  course,  produce  an  evaluation  extremely  dependent  on  the 
evaluator's  subjective  opinion,  experience,  and  consistency  between  the  proto¬ 
type  and  operational  systems. 

8.  Evaluation  Phases.  The  software  operator-machine  Interface  evaluation 
consists  of  four  phases:  planning,  calibration,  assessment,  and  analysis. 

a.  During  the  planning  phase,  the  deputy  for  software  evaluation  and  the 
software  test  manager  Identify  the  operator-machine  Interface  that  Is  to  be 
evaluated,  when  and  where  the  evaluation  Is  to  take  place,  and  who  the  evalua¬ 
tors  (operators)  are  to  be.  They  are  responsible  for  assuring  that  required 
resources  are  available  to  the  operators  at  evaluation  time. 

b.  During  the  calibration  phase,  the  operators  receive  a  detailed 
procedures  briefing  on  use  of  the  questionnaire.  During  this  briefing,  each 
question  In  the  questionnaire  Is  discussed  In  detail.  It  Is  extremely  Important 
that  each  operator  leave  the  briefing  with  a  thorough  understanding  of  each  and 
every  question. 

c.  The  assessment  phase  Is  the  period  of  time  during  which  the  operators 
complete  the  questionnaire.  This  phase  will  require  1  day  or  less  for  each 
operator  per  questionnaire  completed. 
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d.  During  the  analysis  phase,  the  deputy  for  software  evaluation  will 
collect  the  completed  questionnaires,  perform  a  quality  check,  and  forward  the 
answer  sheets  to  HQ  AFTEC  for  data  reduction  and  a  preliminary  analysis  of 
results.  The  results  will  be  returned  to  the  deputy  for  software  evaluation  for 
final  analysis  and  Inclusion  In  operational  test  and  evaluation  reports.  The 
features  of  the  AFTEC  analysis  program  and  an  explanation  of  Its  output  are 
contained  in  AFTEC  Pamphlet  800-2,  volume  2. 

9.  Response  Form. 

a.  The  form  on  which  an  evaluator  records  his  response  to  questions  is 
the  General  Purpose  (NCS)  Answer  Sheet.  To  avoid  confusion  when  the  answers  are 
transferred  to  the  computer,  any  errors  should  be  completely  erased.  Note  that 
the  NCS  answer  sheet  contains  little  explanatory  Information  since  It  was 
designed  to  be  a  general  form  for  use  by  any  group  using  questionnaires.  There 
are  three  blocks  on  the  answer  sheet:  evaluator  name  block,  numerical  identi¬ 
fication  block,  and  evaluator  response  block. 

b.  The  evaluator  name  block  contains  as  a  minimum  the  last  name  of  the 
evaluator;  It  can  also  contain  the  system  name  (alpha  only)  at  the  choice  of  the 
software  test  manager.  The  accuracy  of  this  block  Is  not  as  critical  as  the 
following  two  blocks.  The  suggested  format  with  an  example  Is  given  in  figure 
1. 

c.  The  numerical  Identification  block  contains  numeric  codes  for  the 
questionnaire  type,  four  levels  of  system  Identification,  and  the  evaluator. 
The  numeric  codes  are  entered  In  the  appropriate  column  fields  (A  through  L), 
and  the  associated  numbered  circles  are  darkened.  Extreme  care  should  be  taken 
to  enter  all  data  In  this  block  correctly  since  this  block  is  the  only  output 
Information  which  uniquely  associates  the  evaluator  responses  with  the  correct 
evaluator  and  system.  Do  not  leave  any  blanks  In  Items  A  through  L;  use  zeros 
rather  than  leave  blanks.  The  system  code  (Items  B  and  C)  must  be  supplied  by 
the  software  test  manager.  Additional  system  Identification  should  be  agreed  to 
by  the  deputy  for  software  evaluation  and  the  software  test  manager  and  supplied 
to  each  evaluator.  The  required  format  Is  shown  in  figure  2  with  an  example 
given  In  figure  3.  At  the  option  of  the  software  test  manager,  additional  data 
can  be  entered  In  this  block,  such  as  the  date  of  the  evaluation  (in  the  birth- 
date  field),  or  the  number  of  hours  required  to  complete  the  answer  sheet 
(fields  0  and  P). 

d.  The  evaluator  response  block  Is  filled  in  using  the  following  response 
scale  to  answer  each  question: 

A.  COMPLETELY  AGREE  (absolutely  no  doubt) 

B.  STRONGLY  AGREE 

C.  GENERALLY  AGREE 

D.  GENERALLY  OISAGREE 

E.  STRONGLY  DISAGREE 

F.  COMPLETELY  DISAGREE  (absolutely  no  doubt) 

One  of  these  responses  must  be  given  for  each  question.  In  addition,  one  or 
more  of  the  following  standardized  comment  responses  can  be  selected: 

I.  I  had  difficulty  answering  this  question. 

J.  A  written  comment  has  been  submitted. 
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The  following  example  Is  provided  for  completing  the  Name  Identification  Block 
on  the  General  Purpose  (NCS)  Answer  Sheet  used  with  the  AFTEC  questionnaires. 


NAME  (Lost.  First.  M  l.) 


E i*  mi  ott  Amt  u  o  miv  ..fis 


®®®®®®®®#®®®®®®®®®®® 

®©®©©®©®®®®®®®®®®®®® 

©©©©©©©©©©©©©©©©©©©© 

©©©©©©©©©©©©©©©©©©©© 

•©©©©©©©©©©©©©©©©©©© 

©©©©©©©©©©©©©©©©©•©© 

©©©©©©©©©©©©©©©©©©©© 

®©©©©©©©©©®#©©©©©©®® 

0OQt000O000000O0O0OO 

©©0©©©®®©©©©0®®0©©©© 

®©®®®®®©®®®®®®®®®®®® 

©••©©©©©©©©©©©©©©©©© 

@©©@©®©©©<M>©®®(M>®®©®®® 

®®©@©®®®®#©®®#®©©©®® 


©©©©©©©©©®©©©©©©©©©© 

©@®©@®©©@®@©@@@@@@@© 

®©©©©©©©®©©©©©©©©®©© 


©©©©©••©©©•©©©©©©©©© 
©©©©©©©©©©©©©©©©©©©© 
®©©©©©©©©®©©®®©©©©©© 
®©©©©®©©©©©©©®©©  Qf  ©  ©  © 

®®®®®®®®0®®®®®®®®0®® 

©©©©©©©©©©©©©©#©©©©© 

®©©©©©®®©®©©©©©®®©©® 


NOTE:  The  SEX  block  and  the  GRADE  or  EDUC  block  need  not  be  filled  In. 
Figure  1.  Evaluator  (Operator)  Name  Block  Example 
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The  following  format  Is  required  for  completing  the  Date  and  Numerical 
Identification  Block  on  the  General  Purpose  (NCS)  Answer  Sheet  used  with  the 
AFTEC  questionnaires.  The  correct  numerical  Integers  must  be  written  in  the 
appropriate  fields.  Extreme  care  should  be  taken  in  entering  these  data  and 
In  completely  covering  the  associated  numbered  circle  in  each  column.  This  is 
the  only  way  that  the  questionnaire  response  can  be  correlated  with  the 
system/subsystem/evaluator. . . 


|  BIRTH  PATE _ IDENTIFICATION  NUMBER  SPECIAL  CODES  ; 

'  mo.  |  pay  |  yr.  ~a|b|c|d|eIfIG|h|i|j  k|l|m|n|o|p| 


Columns 


Data  Description 


Birth  Date 

Date  Evaluation  Started 

Mo. 

Jan  -  Dec 

Day 

01-31 

Yr. 

80-99 

A 

Type  of  Questionnaire 

3  (SOMIQ) 

B.C 

System  Code 

01-99 

D,E 

Subsystem  Code 

01-99 

F.G 

Further  system  Identification  if  needed 

01-99 

H.I 

Further  system  identification  if  needed 

01-99 

J.K.L 

Evaluator  Code 

001-999 

M.N 

(not  used) 

(Blank) 

O.P 

Time  (hrs)  to  complete  questions 

00-99 

Figure  2.  Numerical  Identification  Block  Example 

r 


rerv; 
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The  following  example  Is  provided  for  completing  the  Date  and  Numerical 
Identification  Block  on  the  General  Purpose  (NCS)  Answer  Sheet  used  with  the 
AFTEC  questionnaires. 


i  BIRTH  DATE 

IDENTIFICATION  NUMBER 

SPECIAL  CODES 

MO. 

DAY 

VR 

A 

B 

c 

D 

E 

F 

G 

H 

1 

J 

K 

L 

M 

N 

O  P 

Jan.  Q 

0 
War  O 
An.  O 

»•»  O 

jut  Q 
Jul  O 
Au»  O 

Sait  O 
Ocl  O 

Matt  Q 
»«• 

11 

CO 

3 

l 

2 

0 

■5 

0 

v> 

* 

i 

1 

0  t 

(«)  ® 

•  © 

©© 

®© 

• 

© 

© 

© 

© 

© 

w® 

©# 

©  © 
©© 
©© 
©  © 
©  © 
©© 
•  @ 
©© 

©©©•©••••• 

©©©©©©©©©© 

©••©©©©©©© 

•©©©•©©©©© 

©©©©©©©©©© 

®©©©©©®©®® 

©®©©©©©®©© 

©©©©©©©©©© 

®©®®©©®®©© 

©®®©©®®©©© 

•  ©©©•(?) 
©•©GO# 
©©©©©© 
©  ®  ©  ©  ©  © 
©©©©©© 
©©©©©© 
©©©©©© 
©©©©©© 
®  ©  ©  ©  ®  ® 
©©©©©© 

Columns 

Value 

Meaninq 

Birth  Date 

Mo.  Day  Yr. 

14  Dec  81 

Evaluation  conducted  on  14  Dec  1981 

A 

3 

S0MIQ  Evaluation 

B.C 

22 

System  #22  (JTIDS) 

D,E 

03 

Subsystem  #03  (Fault  Isolation 
Software) 

F,G 

00 

not  used 

H.I 

not  used 

J.K.L 

001 

Evaluator  #001  (Elliott) 

0,P 

01 

Elliott  took  1  hour  to  complete 
this  questionnaire  on  FIS 

5- 


Figure  3.  .  meric  Identification  Block  Example 
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The  responses  G  and  H  currently  have  no  meaning.  The  responses  A  to  F  indicate 
the  extent  to  which  the  evaluator  agrees/disagrees  with  the  question  statement. 
Depending  on  the  application  area  and  the  type  of  question,  these  responses  can 
be  Interpreted  differently.  In  general,  however,  the  response  scale  can  be 
Interpreted  as  follows: 

A.  COMPLETELY  AGREE.  There  must  be  absolutely  no  doubt  when  using 
this  response  that  the  product  being  evaluated  cannot  be  any  better  with  respect 
to  the  characteristic  addressed. 

B.  STRONGLY  AGREE.  This  response  indicates  that  the  product  being 
evaluated  is  very  good  and  very  helpful  to  the  software  maintainer. 

C.  GENERALLY  AGREE.  This  response  indicates  that  the  product  being 
evaluated  Is  acceptable  and  helpful  to  the  software  maintainer. 

D.  GENERALLY  DISAGREE.  This  response  Indicates  that,  although  the 
product  being  evaluated  Is  acceptable,  some  improvements  are  required  to  make  it 
helpful  to  the  software  maintainer. 

E.  STRONGLY  DISAGREE.  This  response  indicates  that  the  product 
being  evaluated  Is  unacceptable  and  major  Improvement  is  required  before  it 
would  be  helpful  to  the  software  maintainer. 

F.  COMPLETELY  DISAGREE.  There  must  be  absolutely  no  doubt  when 
using  this  response  that  the  product  being  evaluated  is  unacceptable  and  must  be 
completely  redesigned  or  rewritten  to  be  acceptable  with  respect  to  the  charac¬ 
teristic  addressed.  It  is  emphasized  that  responses  of  A  or  F  are  in  general 
not  expected  since  these  responses  indicate  a  best  possible  or  worst  possible 
characteristic  relative  to  software  in  general. 

e.  Not  applicable.  Occasionally  the  evaluator  will  encounter  a  question 
that  does  not  seem  to  apply  to  the  system  being  evaluated.  In  this  circum¬ 
stance,  the  evaluator  will  have  to  work  harder  at  formulating  an  answer.  When  a 
question  seems  to  be  "not  applicable",  it  Is  usually  because  some  feature  or 
function  of  the  Idealized  systems  is  entirely  absent  in  the  system  under  con¬ 
sideration.  In  this  case,  the  evaluator  must  formulate  an  answer  based  upon  any 
system  deficiency  caused  by  the  lack  of  the  desired  feature.  If  operation  of 
the  system  would  be  enhanced  by  the  presence  of  the  missing  feature,  then  the 
question  should  be  answered  In  the  D,  E  or  F  range.  If,  on  the  other  hand, 
presence  of  the  missing  feature  would  not  help  the  operator,  the  question  should 
be  answered  In  the  A,B,  or  C  range.  The  reason  that  the  answer  in  such  "not 
applicable"  cases  Is  not  always  A  or  F  Is  that  the  missing  feature  may  be  a 
nlce-to-have  Item.  It  is  up  to  the  evaluator  to  judge  how  seriously  lack  of 
certain  features  affect  the  operator's  ability  to  complete  the  mission.  Every 
question  must  have  an  answer  of  A  through  F.  The  deputy  for  software  evaluation 
must  review  each  questionnaire  for  completeness  before  sending  it  to  HQ  AFTEC 
for  data  processing. 

10.  Evaluation  Concepts.  When  performing  the  software  operator-machine  Inter¬ 
face  evaluation,  the  operator  must  keep  several  important  concepts  in  mind: 

a.  Questions.  The  "questions"  In  the  questionnaire  are  not  questions  at 
all.  Each  question  Is  a  positive  statement  about  a  desirable  design  feature  as 
It  would  be  Implemented  In  an  Ideal  system.  The  question  Is  answered  by  the 
evaluator  Indicating  how  well  the  system  under  evaluation  lives  up  to  the  ideal 
described  In  the  question. 
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b.  Keep  the  characteristic  in  mind.  When  answering  a  question  in  a  given 
characteristic  group,  answer  the  question  only  as  It  applies  to  the  definition 
of  that  characteristic.  For  instance,  question  9  (the  data  entry  has  a  cursor 
or  pointer)  under  assurability  should  only  be  answered  as  to  how  effective  the 
display  is  in  telling  the  operator  where  his  next  character  will  be  typed  in  a 
line  of  text,  thereby  avoiding  the  error  of  no  blanks  or  too  many  blanks.  True, 
this  contains  a  certain  amount  of  descriptiveness  (as  do  almost  all  questions), 
but  the  emphasis  here  is  on  assurability  (error  avoidance  in  this  case).  If  the 
evaluator  is  confused  about  the  difference  in  meaning  among  two  or  more 
questions,  he  or  she  can  refer  to  attachment  2  to  find  all  of  the  questions 
related  to  that  subject  and  compare  them  to  one  another. 

c.  Take  your  time.  Heavy  emphasis  must  be  placed  upon  the  best,  most 
carefully  considered  answers  you  can  give.  A  few  extra  minutes  of  careful 
thought  right  now  may  easily  save  the  Air  Force  hundreds  of  hours  of  needless 
hassle  later  on.  System  deficiencies  revealed  now  can  be  corrected  before  the 
system  is  in  field  use. 

d.  Use  the  comment  sheets.  Everything  said  in  the  previous  paragraph 
applies  equally  well  here.  This  questionnaire,  being  a  human  product,  cannot 
possibly  be  expected  to  reveal  every  shortfall  of  every  system.  Your  knowledge 
as  expressed  in  the  comment  sheets  (AFTEC  Form  207)  is  the  only  way  that  this 
analysis  can  reveal  deficiencies  not  specifically  or  exactly  addressed  by 
questions.  An  example  Form  207  is  filled  out  In  figure  4. 

11.  How  to  Recommend  Changes.  Since  it  is  the  considered  opinion  of  AFTEC  that 
this  manual  Is  not  a  "perfect**  test  too?  (nor  will  it  ever  be),  AFTEC  must  be 
prepared  to  change  the  document  as  time  passes.  One  of  the  best  sources  of 
additional  information  to  be  included  in  this  manual  is  the  very  people  who  use 
It.  Therefore,  a  blank  question  data  sheet  has  been  provided  in  figure  5  to  be 
used  as  a  medium  with  which  the  evaluator  (operator)  may  forward  recommended 
changes  to  AFTEC.  The  question  data  sheet  may  be  used  to  address  exact  ques¬ 
tions  (fill  in  the  question  number)  or  to  suggest  new  questions.  Please  send 
the  question  data  sheet  along  with  any  additional  information  to: 

HQ  AFTEC/LG5 

Klrtland  AFB,  NM  87117 

Please  submit  a  Computer  Program  Observation  Report  (AFTEC  Form  207)  with  the 
question  data  sheet  to  identify  yourself  and  the  circumstances  which  lead  to 
your  recommendation.  AFTEC  personnel  will  contact  you  and  discuss  the  recom¬ 
mendation  with  you. 
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COMPUTER  PROGRAM  OBSERVATION  REPORT 


SYSTEM  M-7.  B»S2G. 
4ITL) 


SUB-SYST  EM 'COMPUTER  (Nav 

COMPUTER  PROGRAM  (OFF.  Aerodynamic 

ay  atom t.  Fire  Control  Computer) 

Date  Update  Program) 

Checkout  Station 

User's  Manual 

NAME  (Laat,  Fit  at.  Mid  tit*  Initial) 

Kelly,  Robert  J. 


ORIGINATOR  IQENTIf  tCATIQN  flnTt»«|Mof,  coordinator,  approving  ollidola,  etc) 
l.  Middle  Initial)  I  GRAOE  I  ORGANIZATION  AND  STATION 


SU  0  -PROGR  AM  MODULE 
(7m  hide  van  ton  designa¬ 
tion 


DUTY  PHONE 

21*4-91*21 


IH. _ OTHER  DATA 

DATE  AND  TIME  OF  OBSERVATION  f|f  I  assort  ant.  Include  etapaed  ttmea) 

_ 23-Juo  82 _ 


DOCUMENT  NO. 


DOCUMENT  REFERENCES 
RUBUSHER<USAF.  Rand  Com, 


DEFICIENCY  REPORT  waTCm  ITEM 
I  !  YES  NO 


SR  AGE  NO. |  FIGURE 


IV.  ORSIRVATIQN  (Include  what  did/did  not  happen,  what  ahould  Roy  happened,  reaulta,  au Mjeated  c hang* a,  relevant  condlttonm,  etc) 

SOMIQ  QUESTION  #  UU 

The  COS  user's  manual  (D290-70021-1)  Is  very  poorly  organized.  System  menu  pages 
could  be  referenced  differently  to  help  operator  more  easily  located  different 
pages.  Currently  there  is  no  sequence  to  the  menu/page  #s  in  the  procedure. 


.u°~.  207 


Figure  4.  Sample  Form  207 
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QUESTION: 


CHARACTERISTIC: 


EXPLANATION: 


EXAMPLES: 


GLOSSARY: 


QUESTION  DATA  SHEET 


Question  Number 


(Use  this  sheet  to  recomment  new  questions  or  changes  to  existing  questions  to 
AFTEC) 


Figure  5.  Question  Data  Sheet 
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Commander 


ARTHUR  K.  KLEMP ,/fla j ,  USAF 
Director  of  Administration 
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ATTACHMENT  1 

QUESTION  RESPONSE  GUIDELINES 


The  following  sections  contain  Information  which  should  help  clarify  the 
Intent  of  each  question  to  the  evaluator.  There  Is  one  section  per  charac¬ 
teristic.  Each  page  within  each  section  corresponds  to  a  question  from  the 
Software  Operator-Machine  Interface  Questionnaire.  Many  questions  have  special 
response  Instructions  which  should  be  reviewed. 
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A.  ASSURABILITY.  A  computerized  system  contains  the  quality  of  assurability 
to  the  extent  that  it  aids  the  operator  in  validating  data,  avoiding  errors,  and 
correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error  avoidance  may 
or  may  not  achieve  this  goal.  A  system  should  also  be  designed  so  that  errors 
are  easy  to  correct,  and  above  all  so  that  errors  do  not  have  catastrophic 
effects. 
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QUESTION  DATA  SHEET 


Question  Number  1 


QUESTION:  Operator  input  errors  do  not  cause  system  failures. 

CHARACTERISTIC:  Assurabil Ity. 

EXPLANATIONS:  Commands  and  data  entered  by  the  operator  should  not  cause 
hardware  or  software  errors  inside  the  machine. 

EXAMPLES:  A  bad  command  by  the  operator  could  be  a  request  to  the  computer  to 
connect  a  12-VDC  power  supply  to  the  same  bus  that  a  5-VDC  power  supply  Is 
connected.  Another  bad  command  by  the  operator  could  be  entering  an  "8"  when 
the  computer  could  be  requesting  a  number  between  "1"  and  "7."  Other  bad 
commands  could  be  instructions  to  start  a  device  which  is  already  running  or 
stop  one  which  is  already  stopped.  The  reason  that  these  are  not  "don't  care" 
situations  is  that  the  operator  may  have  intended  to  start  or  stop  a  different 
device. 

GLOSSARY:  Operator  input:  Data  or  commands  entered  as  typewritten  text, 
pushbutton  sequences,  control  stick  movements,  card  decks,  tapes,  light  pens, 
etc. 
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QUESTION  DATA  SHEET 

Question  Number  2 


QUESTION;  Operator  Input  errors  are  detected. 

CHARACTERISTIC:  Assurabllity. 

EXPLANATIONS:  The  machine  must  be  designed  to  detect  operator  input  errors  or 
it  will  try  to  function  using  bad  data.  Anything  the  machine  does  based  on  the 
bad  data  may  be  wrong,  as  well  as  possibly  dangerous  to  both  equipment  and 
personnel . 
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QUESTION  DATA  SHEET 
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Question  Number  3 


QUESTION:  The  causes  of  input  errors  are  displayed  to  the  operator. 
CHARACTERISTIC:  Assurablllty. 

EXPLANATIONS :  When  an  operator  has  made  an  error,  the  operator  needs  to  know 
what  the  error  was  in  order  to  correct  it. 


EXAMPLES:  It  Is  insufficient  for  a  machine  to  display  some  message  such  as 
"Input  error".  Such  a  message  does  not  tell  the  operator  what  was  wrong,  only 
that  something  was  wrong. 
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QUESTION  DATA  SHEET 

Question  Number  4 


QUESTION:  The  action  required  to  correct  an  operator  input  error  Is  displayed 
to  the  operator. 

CHARACTERISTIC:  Assurabl 1 Ity . 

EXPLANATIONS:  Having  the  machine  tell  the  operator  what  to  do  to  correct  an 
error  Is  much  quicker  than  having  the  operator  refer  to  manuals,  etc.,  to  look 
for  the  proper  procedure. 

EXAMPLES:  The  machine:  ENTER  DATE 
The  operator:  Jun  16,  1980 
The  machine:  NO,  FORMAT:  YY,  MM,  DO 
The  operator:  80,  06,  16 
The  machine:  OK 
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QUESTION  DATA  SHEET 


QUESTION:  Input  errors  are  easily  corrected. 
CHARACTERISTIC:  Assurabi 1 ity. 


Question  Number  5 


EXPLANATIONS:  The  actions  necessary  to  correct  errors  can  vary  all  the  way  from 
completely  reinitializing  the  system  to  simply  correcting  the  erroneous  portion 
of  a  command.  The  correction  of  operator  errors  should  be  as  simple,  easy,  and 
straightforward  as  possible. 

EXAMPLES:  If  the  operator  enters  a  long  command,  such  as  "Load  Program 
TESTLRU17  frm  disk  22,"  the  machine  should  allow  the  operator  to  correct  a 
misspelling  easily  rather  than  retype  the  entire  command  sequence. 

The  machine  would  redisplay  the  Incorrect  command  (above)  and  allow  the 
operator  to  Insert  the  "o"  In  the  word  "frm"  to  create  the  word  "from,"  and  then 
accept  the  corrected  command. 
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QUESTION  DATA  SHEET 


Question  Number  6 


QUESTION:  Input  errors  are  quickly  corrected. 

CHARACTERISTIC:  Assurabillty. 

EXPLANATIONS:  For  an  error  to  be  termed  "quickly  correctable,"  the  system  must 
resume  normal  operation  immediately  after  the  error  is  corrected. 
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QUESTION  DATA  SHEET 


Question  Number  7 


QUESTION:  The  operator  can  verify  input  before  execution/entry. 

CHARACTERISTIC:  Assurabil ity. 

EXPLANATIONS:  If  the  operator  can  see  the  instruction  as  it  is  entered,  then 
mistyped  characters,  etc.,  can  be  corrected  on-the-spot  rather  than  after  the 
machine  indicates  Incorrect  input. 
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QUESTION  DATA  SHEET 


Question  Number  8 


QUESTION:  Mission-peculiar  data  entered  by  the  operator  are  checked  for  vali¬ 
dity. 

CHARACTERISTIC:  Assurabil ity. 

EXPLANATIONS:  When  the  operator  enters  mission  data  (altitude,  voltage,  range) 
the  data  should  be  checked  by  the  machine  as  much  as  feasible  to  ensure  that  the 
mission  Is  not  completed  with  wrong  data,  resulting  in  reaccomplishment. 

EXAMPLES:  An  aircraft  simulator  operator  enters  a  simulated  terrain  altitude  of 
60,000  feet  when  it  should  have  been  6,000  feet,  thereby,  causing  a  "crash"  and 
subsequent  restart  of  the  simulation.  The  data  should  have  been  checked  against 
a  maximum  of,  approximately  26,000  feet. 

ADDITIONAL  EXAMPLES: 

ERRONEOUS  ENTRY  (SHOULD  HAVE  BEEN) 

100  degrees  North  latitude  (10.0  N  Lat) 

32  Jan  80  (31  Jan  80) 

Mach  12  (Mach  1.2) 

GLOSSARY :  Mission  peculiar:  any  data  which  are  entered  by  the  operator  as  a 
part  of  normal  operations  but  which  may  change  mission-to-misslon  or  task-to- 
task. 
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QUESTION  DATA  SHEET 

Question  Number  9 


QUESTION:  The  data  entry  display  has  a  cursor  or  pointer. 

CHARACTERISTIC:  Assurabil ity. 

EXPLANATIONS:  The  operator  must  know  where  the  next  character  entered  will  be 
placed  to  avoid  errors  caused  by  too  many  spaces  or  "double"  errors  caused  by 
correcting  the  wrong  character. 

EXAMPLES :  The  operator. enters  "REWIIO  TAPE  22,"  then  notices  the  error.  The 
system  allows  him  to  backspace  and  change  "REWIID"  to  "REWIND."  A  cursor  allows 
him  to  backspace  quickly  and  accurately.  Lack  of  a  cursor  causes  him  to  count 
backspaces  and  perhaps  change  "REWIIO"  to  "REWNID." 

The  communications  system  operator  in  an  E-4B  is  entering  a  command  when 
the  aircraft  suddenly  hits  turbulence  and  the  operator's  finger  hits  the  space 
bar  several  times.  With  improper  spacing,  the  command  will  be  formatted  incor¬ 
rectly  and  the  machine  will  reject  the  command,  forcing  the  operator  to  re¬ 
enter.  A  cursor  would  allow  the  operator  to  see  how  many  extra  spaces  were 
entered,  enter  the  appropriate  number  of  backspaces,  and  resume  entry  where  it 
was  interrupted. 

GLOSSARY :  Cursor  or  pointer:  a  method  of  indicating  on  the  display  where  the 
next  character  to  be  entered  will  appear  on  the  display. 
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QUESTION  DATA  SHEET 

Question  Number  10 

QUESTION:  The  operator  is  able  to  correct  mistyped  characters  with  a  backspace 
key. 

CHARACTERISTIC:  Assurabil ity. 

EXPLANATIONS:  It  is  much  faster  and  less  error  prone  to  be  able  to  backspace 
and  change  an  incorrect  character  than  it  is  to  be  forced  to  retype  an  entire 
entry. 

GLOSSARY :  Backspace:  rubout,  back  arrow,  etc. 
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QUESTION  DATA  SHEET 

Question  Number  11 


QUESTION:  The  system  does  not  require  the  operator  to  copy  information  by  hand. 

CHARACTERISTIC:  Assurability. 

EXPLANATIONS:  Hand  copying  information  from  a  computer  display  to  some  form  of 
record  is  a  source  of  errors  and  extra  work  which  could  be  handled  by  the 
machine. 

EXAMPLES:  An  operator  is  displaying  satellite  telemetry  points  on  a  screen  and 
desires  some  form  of  hardcopy.  If  the  operator  is  forced  to  copy  the  infor¬ 
mation  by  hand  as  the  data  are  changing,  the  data  are  useless  since  the  numbers 
copied  down  will  not  all  be  taken  at  the  same  time.  The  operator  should  have 
some  kind  of  "picture  taking"  capability  which  would  automatically  produce 
printed  forms  of  the  data. 
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QUESTION  DATA  SHEET 


Question  Number  12 


QUESTION:  Task,  aborts  and  interrupts  do  not  have  detrimental  side  effects. 
CHARACTERISTIC:  Assurability. 

EXPLANATIONS:  Turning  the  machine  off  for  an  emergency  power  failure  or  stop- 
ping  a  test,  etc.,  should  not  result  in  mechanical  damage  or  loss  of  data  files 
which  would  be  difficult  to  replace. 

EXAMPLES:  The  machine  is  performing  a  test  from  a  tape  drive.  Power  failure 
causes  the  tape  drive  to  tear  the  tape  in  half.  The  tape  can  only  be  replaced 
by  a  new  tape  from  the  depot. 

GLOSSARY: 

Task  aborts:  emergency  shutdown;  (simulator)  mission  restart;  and  a 
variety  of  other  reasons  for  wanting  to  simply  quit  at  any  time  other  than  when 
the  machine  would  normally  expect. 

Task  interrupt:  pause,  freeze,  or  even  "hold  it — it's  time  for  my  coffee 
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QUESTION  DATA  SHEET 


Question  Number  13 


QUESTION:  Selecting  a  device  off-line  does  not  have  detrimental  side  effects. 
CHARACTERISTIC:  Assurabl 1 Ity. 

EXPLANATIONS:  When  an  operator  takes  a  device  off-line,  the  action  should  not 
interrupt  system  operation  as  It  pertains  to  that  device. 

EXAMPLES:  An  operator  is  waiting  for  a  long  test  to  be  completed.  The  printer 
is  not  In  use,  but  the  operator  knows  it  Is  short  of  paper.  Taking  the  printer 
off-line  to  replace  the  paper  should  not  cause  the  test  to  stop. 

A  weapon  system  operator  shuts  off  one  display  scope  which  is  obviously 
malfunctioning  and  is  distracting  him.  Shutting  this  one  scope  off  should  not 
cause  any  other  changes  to  the  system,  and  should  certainly  not  cause  the  whole 
electronic  warfare  system  to  quit. 

GLOSSARY :  Off-line:  out  of  operation,  power  off,  or  de-selection. 
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QUESTION  DATA  SHEET 

Question  Number  14 


QUESTION:  The  system  automatically  ceases  execution  If  Internal  errors  are 
detected. 

CHARACTERISTIC:  Assurablllty. 

EXPLANATIONS:  When  the  computer  detects  an  error,  It  should  Immediately  halt 
all  executions  to  avoid  damage  to  data  files  and  to  avoid  giving  wrong  answers. 

EXAMPLES:  Data  words  stored  In  the  computer  Internal  storage  represent  altitude 
above  ground  level.  When  recalled,  the  words  had  a  read  or  parity  error.  If 
the  words  were  Interpreted  and  sent  to  the  pilot  It  might  tell  him  2000  feet, 
whereas,  the  correct  altitude  might  be  50  feet. 

GLOSSARY :  Parity:  a  method  of  checking  the  validity  of  data.  A  parity  error 
Indicates  Invalid  data. 
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QUESTION  DATA  SHEET 


Question  Number  15 


QUESTION:  The  operator  Is  alerted  to  faults  within  the  system. 

CHARACTERISTIC:  Assurabi 1 ity . 

EXPLANATIONS:  This  question  is  aimed  at  eliminating  operator  work,  having  to  be 
redone  once  the  fault  Is  discovered.  The  software  should  be  designed  to  detect 
as  many  failures  as  possible  and  to  report  them  to  the  operator  as  soon  as 
possible.  Any  unscheduled  change  In  the  status  of  any  part  of  the  system  should 
be  reported  to  the  operator. 

EXAMPLES: 

1.  The  operator  Is  entering  a  long  sequence  of  instructions  and  does  not 
realize  the  machine  Is  In  an  unusable  status.  When  he  finally  finds  out  the 
machine  Is  "broken,"  he  must  turn  It  off,  repair  It,  then  reenter  the  long 
sequence  of  commands. 

2.  It  Is  very  nice  for  a  pilot  to  know  when  the  terrain  avoidance  radar 
ceases  to  function  or,  even  worse,  starts  to  give  erroneous  data. 

GLOSSARY: 

Diagnostic:  fault  detection  and  reporting  hardware. 

Faults:  burned  out  power  supply,  out-of-calibration  stimulus  voltage,  etc. 
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Question  Number  16 


QUESTION:  The  causes  of  system  halts  are  displayed  to  the  operator. 
CHARACTERISTIC:  Assurabllity. 

EXPLANATIONS:  This  Is  Important  Information  for  anyone  who  must  fix  or  restart 
the  system.  When  the  system  halts,  the  cause  should  be  displayed  even  If  the 
cause  may  not  be  operator  correctable. 

EXAMPLES:  An  aircraft  pilot  needs  to  know  whether  the  navigation  computer  shut 
Itself  down  due  to  equipment  bay  overheat,  faulty  programming,  or  just  random 
error.  Restarting  the  computer  In  a  hot  compartment  could  damage  the  equipment. 
Restarting  the  computer  to  execute  a  scrambled  program  would  be  useless  or 
possibly  dangerous.  Restarting  the  computer  after  a  random  error  would  probably 
be  OK. 
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ASSURABILITY.  A  computerized  system  contains  the  quality  of  assurability  to  the 
extent  that  It  aids  the  operator  in  validating  data,  avoiding  errors,  and 
correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error  avoidance  may 
or  may  not  achieve  this  goal.  A  system  should  also  be  designed  so  that  errors 
are  easy  to  correct,  and  above  all  so  that  errors  do  not  have  catastrophic 
effects. 
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B.  CONTROLLABILITY.  A  computerized  system  contains  the  quality  of  controll¬ 
ability  to  the  extent  that  It  allows  the  operator  to  direct  the  operations  of 
the  machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of  the  machine 
in  order  to  utilize  it  effectively  and  efficiently. 
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QUESTION  DATA  SHEET 


Question  Number  17 


QUESTION:  The  operator  can  interrupt  and  resume  automatic  processes. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  It  is  often  desirable  for  the  operator  to  halt  execution  of  a 
process,  change  some  parameters,  and  then  proceed  with  the  process. 

EXAMPLES:  A  test  equipment  operator  notices  that  while  both  a  voltage  and 
resistance  measurement  are  within  tolerance,  the  voltage  is  borderline  high  and 
the  resistance  is  borderline  low.  The  operator  interrupts  automatic  test, 
calculates  current,  and  sees  that  It  is  actually  still  within  tolerance. 
The  automatic  test  is  then  resumed. 
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QUESTION  DATA  SHEET 


Question  Number  18 


3UESTI0N:  The  operator  has  task-abort  capabilities  available. 
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CHARACTERISTIC  Controllability. 

EXPLANATIONS:  It  is  often  necessary  to  stop  a  task  before  it  is  completed 
because  of  time  constraints,  errors,  or  higher-priority  workload. 

EXAMPLES:  A  mission  preparation  operator  is  printing  out  a  table  of  latitudes 
and  longitudes  when  he  notices  that  the  first  entry  is  in  error.  He  must  wait 
for  the  entire  table  to  print  out  before  he  can  correct  the  error  and  reprint 
because  there  Is  no  abort  capability.  If  he  had  an  abort  capability  he  could 
abort  the  print,  fix  the  error,  and  then  restart  the  print  immediately. 
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QUESTION  DATA  SHEET 


Question  Number  19 


QUESTION:  The  operator  may  initiate  selected  self-test  systems. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Automated  troubleshooting  procedures  are  a  great  help  to  opera 
tors,  but  not  all  of  them  may  be  used  continuously.  Therefore,  some  of  them 
must  be  initiated  by  the  operator  at  times  when  they  would  not  interfere  with 
normal  system  operation. 

EXAMPLES:  The  self-test  systems  may  be  as  simple  as  lightbulb  checks  or  display 
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QUESTION  DATA  SHEET 


Question  Number  20 


QUESTION:  The  operator  may  send  output  data  to  various  devices. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Giving  the  operator  control  over  where  to  send  output  data  allows 
the  operator  greater  flexibility  in  completing  the  mission. 

EXAMPLES: 

1.  An  aircrew  training  device  (ATD)  operator  can  control  which  scope  can 
display  which  Information.  Hardcopies  can  also  be  generated. 

2.  An  automatic  test  station  has  a  printer  which  is  out  of  paper.  If  the 
operator  can  divert  the  printed  output  to  the  display  screen,  the  necessary 
repair  information  can  at  least  be  copied  down  by  hand  and  the  item  can  be 
fixed.  Otherwise,  maintenance  activities  cease  until  paper  for  the  printer  is 
procured. 
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QUESTION  DATA  SHEET 


Question  Number  21 


QUESTION:  The  operator  can  ask  for  and  receive  the  current  status  of  operation. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Especially  on  long  tasks,  it  is  necessary  for  the  operator  to 
know  that  things  are  proceeding  as  they  should.  Otherwise,  the  operator  might 
wait  until  the  end  of  a  2-hour  task  only  to  find  out  that  it  has  to  be  redone. 

EXAMPLES: 

1.  A  test  equipment  operator  is  performing  a  6-hour  alignment  test  on  an 
inertial  platform.  After  15  minutes,  he  queries  the  test  equipment  system 
status  and  sees  that  the  machine  has  been  doing  nothing  but  trying  to  connect  a 
power  supply  which  is  not  turned  on.  Since  he  could  query  the  system  status,  he 
discovered  the  situation  almost  immediately  rather  than  6  hours  later. 
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QUESTION  DATA  SHEET 


Question  Number  22 


QUESTION:  The  operator  can  select  the  type  of  information  shown  on  the  display. 
CHARACTERISTICS:  Controllability. 

EXPLANATIONS:  The  operator  must  be  able  to  control  what  type  of  information  is 
displayed  so  that  desired  information  may  be  obtained  in  as  easily  understand¬ 
able  a  method  as  possible. 

EXAMPLES: 

1.  A  student  pilot  is  flying  his  aircraft  on  a  strafing  run.  There  are 
so  many  arrows,  lines,  and  indicators  on  his  head-up  display  that  he  is  becoming 
confused.  He  tabs  the  "declutter"  button  and  notices  immediately  that  a  "low 
airspeed"  warning  had  been  hidden  amongst  all  the  garbage.  He  corrects  the 
situation  and  completes  the  mission. 

2.  A  satellite  ground  controller  notices  from  the  satellite  telemetry 
display  that  the  satellite  battery  voltage  is  decreasing.  He  orders  the  battery 
temperature  data  to  also  appear  on  this  display  to  see  if  any  correlation 
exists. 
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QUESTION  DATA  SHEET 


Question  Number  23 


QUESTION:  The  operator  may  command  various  displays  of  system  status. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  operator  must  often  troubleshoot  problems  to  correct  bad 
situations.  To  do  this,  detailed  information  on  the  status  of  various  systems 
and  subsystems  must  be  available. 

EXAMPLES: 

1.  A  data  processing  computer  operator  notices  that  his  system  has  seem¬ 
ingly  stopped.  A  system  status  display  shows  everything  normal.  A  peripheral 
status  display  shows  nothing  wrong.  A  job  control  display  shows  that  one  small 
computer  program  is  using  all  the  system  time.  A  display  of  the  program  status 
log  shows  no  input  and  no  output.  The  operator  determines  that  a  novice  pro¬ 
grammer  has  written  another  infinite  loop;  he  aborts  the  program,  and  the  normal 
processing  of  paychecks  resumes. 

2.  At  a  change  of  shift  on  a  weapon  control  system,  the  new  operator  would 
normally  want  to  display  the  current  status  of  all  sensors,  communications 
lines,  system  armament  resources,  etc. 
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QUESTION  DATA  SHEET 


Question  Number  24 


QUESTION:  The  operator  may  control  the  display  text  that  explains  what  input  is 
needed  or  explains  the  output. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Explanatory  text  is  often  helpful  to  a  beginning  operator,  but 
it  is  just  extra  trash  on  the  display  to  an  experienced  operator. 

EXAMPLES:  One  system  has  two  commands  which  control  this  feature.  The  command 
"VERBOSE"  tells  the  machine  to  display  information  with  plenty  of  explanatory 
text.  The  command  "CONCISE"  tells  the  machine  to  display  the  bare  essentials. 
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QUESTION  DATA  SHEET 

Question  Number  25 


QUESTION:  The  operator  may  edit  the  data  base  prior  to  use  by  the  system. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Many  systems  have  a  "scratchpad"  data  base  which  can  be  edited  by 
the  operator  to  create  a  data  base  which  applies  to  the  current  task. 

EXAMPLES: 

Mission  Data  Preparation:  Prepare  a  bombing  run  with  standard  ingress  and 
egress  route,  changing  the  target. 

Aircrew  Training  Device:  Prepare  a  mission  scenario  involving  an  air 
Intercept  using  nonstandard  differential  altitude. 

Automatic  Test  Equipment:  Change  some  test  parameters  to  reflect  that  the 
line-replaceable  unit  undergoing  test  is  a  specially  modified  box. 

GLOSSARY:  Data  base:  a  set  of  parameters  used  by  a  computer  program  as  opposed 
to  the  computer  program  Itself. 
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QUESTION  DATA  SHEET 


Question  Number  26 


QUESTION:  The  operator  may  create  and  execute  strings  of  commands  as  a  single 
command. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  operator  must  often  perform  repetitive  tasks.  It  is  desir¬ 
able  that  the  operator  be  able  to  reduce  the  workload  of  repetitious  tasks  as 
much  as  possible. 

EXAMPLES: 

1.  The  operator  must  perform  a  series  of  10  instructions  a  great  number  of 
times.  If  he  can  create  a  file  containing  the  10  instructions  and  then  use  1 
Instruction  to  execute  the  entry  file,  his  workload  is  reduced  by  a  factor  of 
10. 

2.  So  called  "function  keys"  are  an  example  of  built-in  work  reduction 
techniques.  If  the  operator  can  program  the  function  keys  to  perform  a  specific 
task,  then  the  operator  has  the  desired  control. 
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QUESTION  DATA  SHEET 


Question  Number  27 


QUESTION:  The  operator  can  command  stepped  execution  of  automatic  processes. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  capability  for  stepped  execution  of  automatic  processes  gives 
the  operator  detailed  information  about  what  is  occurring  during  a  normally 
automatic  process. 

EXAMPLES: 

TT~  An  automatic  test  equipment  operator  is  having  trouble  with  a  sup¬ 
posedly  "good"  black  box  failing  an  automatic  test.  He  reruns  the  test  in 
stepped-execution  mode  which  gives  him  time  to  read  every  action  the  machine 
performs.  He  sees  that  one  part  of  the  test  measures  a  voltage  at  5.2  volts. 
The  test  program  fails  the  box,  stating  the  voltage  should  be  5.0  ±  .1  volts. 
Documentation  states  the  voltage  should  be  5.0  ±  .25  volts.  The  box  is  good, 
the  test  program  is  wrong. 

2.  A  similar  visibility  function  within  an  aircraft  cockpit  would  be  the 
capability  provided  by  the  gun  camera.  Within  an  Aircrew  Training  Device  it 
would  be  videotape  replay. 

GLOSSARY :  Stepped  execution:  computer  provides  output  (numerical  value, 
status,  etc.)  after  each  intermediate  step  of  the  calculations  rather  than  only 
an  output  when  all  of  the  steps  in  the  calculations  are  complete. 
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Question  Number  28 


QUESTION :  The  operator  may  command  different  modes  of  operation. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  The  capability  to  command  different  modes  of  operation  gives  the 
operator  the  ability  to  use  the  machine  in  a  manner  consistent  with  his  need  and 
experience  level . 

EXAMPLES: 

1.  Aircraft:  The  ability  to  drop  bombs  in  single,  pairs,  or  ripple  mode 
as  the  pilot  deems  appropriate. 

2.  Aircrew  Training  Device  (ATP):  The  capability  to  operate  two 

side-by-side  simulators  as  opposing  forces,  one  complete  crew,  or  as  entirely 
separate  ATDs. 

3.  Automatic  Test  Equipment:  Automatic  versus  stepped  execution  testing. 

4.  Missile  Warning  System:  The  capability  to  transmit  messages  auto¬ 
matically  or  manually. 
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QUESTION  DATA  SHEET 


Question  Number  29 


QUESTION:  The  operator  may  control  the  type  and  quantity  of  output. 
CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  Control  of  type  and  quantity  of  output  gives  the  operator  control 
over  what  the  operator  must  read.  Too  much  output,  or  output  the  operator 
doesn't  even  want  to  see,  just  slows  things  down. 

EXAMPLES: 

1.  Under  normal  conditions  the  analyst  may  desire  to  watch  the  console 
display  and  shut  off  the  printer. 

2.  In  order  to  concentrate  on  Important  parameters  during  a  satellite 
maneuver,  the  analyst  can  selectively  prevent  some  of  the  other  parameters  from 
printing  on  the  CRT  during  the  maneuver. 
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Question  Number  30 


QUESTION:  Bypass  procedures  are  available  so  that  in  cases  of  partial  system 
failure  the  more  important  system  functions  can  still  be  performed. 

CHARACTERISTIC:  Controllability. 

EXPLANATIONS:  It  is  often  desirable  to  get  the  job  done  even  though  the  usual 
tools  are  not  available.  This  question  refers  to  degraded  modes  of  operation. 

EXAMPLES: 

1.  An  automatic  test  equipment  operator  sees  that  his  test  has  halted 

because  the  "test  station  fault"  light  is  on.  Researching  the  problem,  he  sees 

that  the  test  program  was  requesting  the  B5  power  supply  be  connected  to  the 
unit  under  test.  The  B5  power  supply  was  sent  to  PMEL  the  previous  week.  He 
causes  the  test  program  to  substitute  the  B18  power  supply  which  can  do  the  same 
job  as  the  B5.  The  test  proceeds. 

2.  A  simulator  has  a  electronic  warfare  office  (EWO)  station.  The  EWO 

station  Is  down  for  repair.  It  is  highly  desirable  to  be  able  to  continue 

training  of  other  crew  members. 

3.  Aircraft:  It  sure  is  nice  to  be  able  to  fly  home  even  if  the  naviga¬ 
tional  computer  quits. 
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CONTROLLABILITY.  A  computerized  system  contains  the  quality  of  controllability 
to  the  extent  that  it  allows  the  operator  to  direct  the  operations  of  the 
machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of  the  machine 
in  order  to  utilize  it  effectively  and  efficiently. 
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C.  WORKLOAD  REASONABILITY.  A  computerized  system  contains  the  quality  of 
workload  reasonability  to  the  extent  that  the  tasks  required  of  the  operator  are 
within  the  operator's  capabilities  and  the  extent  to  which  the  operator  performs 
a  useful,  meaningful  role. 

An  optimum  design  of  a  system  which  involves  an  operator  and  a  computerized 
machine  takes  advantage  of  the  best  capabilities  of  both:  the  machine  to 
perform  repetitive  tasks  rapidly  and  the  operator  to  make  command  decisions 
Involving  unusual  situations. 
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Question  Number  31 


QUESTION:  It  is  easy  to  enter  mission  (task)  peculiar  data. 

CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  Data  must  often  be  added,  revised,  or  changed  to  accomplish  the 
mission.  To  do  this  as  efficiently  as  possible,  the  operator  must  be  given 
control  over  selecting  the  most  efficient  manner  to  perform  the  task. 

EXAMPLES:  A  strike  planning  officer  must  enter  150  coordinates  in  a  table. 
Each  coordinate  must  be  preceded  by  the  coordinate  number;  latitudes  and  long¬ 
itudes  must  be  entered  with  East  or  West,  North  or  South,  including  degree  signs 
and  decimal  points.  This  procedure  is  a  time  consuming  task.  He  selects 
"automatic"  mode  to  speed  the  task.  Coordinate  numbers  are  now  automatically 
sequenced,  all  latitudes  are  automatically  "East"  and  longitudes  are  "North." 
The  machine  enters  the  degree  signs  and  decimal  points  for  him. 
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Question  Number  32 


QUESTION:  Data  preparation  is  usually  performed  using  on-line  devices. 
CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  Preparation  of  data  using  off-line  devices  results  in  using  some 
transfer  medium  (such  as  punch  cards  or  magnetic  tape)  to  transfer  the  prepared 
data  from  the  off-line  device  to  the  computer.  This  procedure  increases  the 
workload. 

GLOSSARY: 

On-line:  connected  directly  to  the  computer  system. 

Data  preparation:  Any  entry  or  changing  of  numbers,  malfunctions,  etc., 
performed  by  the  operator. 

Special  Response  Instructions:  For  an  aircraft  mission  data  base  this  is 
not  a  desirable  feature.  Data  preparation  should  definitely  be  done  away  from 
the  aircraft,  probably  in  the  strike  planning  shop. 
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QUESTION  DATA  SHEET 


Question  Number  33 


QUESTION:  The  system  will  accept  free-format  commands  and  data. 

CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  Acceptance  of  free-format  commands  and  data  decreases  operator 
workload  by  decreasing  the  requirement  for  the  operator  to  meet  rigid,  inflex¬ 
ible  formatting  standards. 

EXAMPLES:  A  voltage,  hours  to  destination,  or  angle  of  attack  value  (e.g., 
five)  that  the  operator  must  enter  is  acceptable  in  any  of  the  following  for¬ 
mats,  including  leading  and  trailing  blanks:  "5",  "5."  "5.0",  "5.00",  "05", 
"05.00",  "0.5  x  10**1",  " . 5E+01" ,  "5D+00." 

GLOSSARY:  Free-format:  any  legitimate  representation  is  acceptable. 
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QUESTION  DATA  SHEET 


Question  Number  34 


QUESTION:  Menu  techniques  are  used  to  aid  the  operator  in  making  decisions. 
CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Having  the  machine  list  possible  alternatives  to  the  operator 
when  an  operator  decision  is- called  for,  aids  the  operator  in  making  a  quick 
decision  and  in  giving  consideration  to  each  alternative.  The  operator  would 
not  forget  to  consider  seldom  used  functions. 

EXAMPLES:  Weapons  Selection: 

A.  Single  0. Slick 

B.  Pairs  E.HI-Drag 

C.  Ripple  F.LO-Drag 


GLOSSARY :  Menu:  a  list  of  items  displayed  on  a  console  display. 
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Question  Number  35 


QUESTION:  The  system  may  be  operated  without  reference  to  manuals  during  normal 
operations. 

CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  An  experienced  operator  will  be  slowed  down  if  forced  to  make 
constant  reference  to  manuals  while  performing  tasks.  Machines  can  be  designed 
to  include  appropriate  manual -type  information  in  the  displays  presented  to  the 
operator. 
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QUESTION  OATA  SHEET 


Question  Number  36 


QUESTION:  The  operator  needs  to  know  a  reasonable  number  of  commands  to  effec¬ 
tively  operate  the  system. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  The  larger  the  number  of  commands  an  operator  must  memorize  in 
order  to  use  the  machine,  the  more  difficult  and  the  more  error-prone  the 
process. 

EXAMPLES:  Operators  which  need  to  use  thousands  of  different  commands  (such  as 
programmers)  must  make  constant  reference  to  manuals.  This  procedure  is  a  time 
consuming  task. 
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QUESTION  DATA  SHEET 


Question  Number  37 


QUESTION:  Messages  to  the  operator  are  easy  to  understand. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS: 

EXAMPLES:  "Ratio  of  actual  speed  to  speed  of  sound"  is  much  more  difficult  to 
understand  than  "Mach  Number." 
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QUESTION  DATA  SHEET 


Question  Number  38 


QUESTION;  The  device  used  to  send  messages  to  the  operator  provides  information 
at  a  rate  comfortable  to  the  operator. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS;  A  display  device  which  sends  information  to  the  operator  too  fast 
can  cause  him  to  miss  Important  information,  possibly  resulting  in  task  reac¬ 
complishment  and  frustration.  A  display  device  which  is  too  slow  causes  the 
operator  to  waste  a  lot  of  time  and  is  also  frustrating. 
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QUESTION  DATA  SHEET 


Question  Number  39 


QUESTION:  The  amount  of  data  presented  to  the  operator  at  one  time  is 
appropriate. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS :  Displaying  too  much  information  to  the  operator  at  one  time  does 
nothing  more  than  add  a  confusion  factor  and  makes  the  operator  work  harder  to 
get  exactly  the  information  desired. 

EXAMPLES:  When  a  radar  system  technician  wants  nothing  more  than  to  know 
antenna  power  output,  it  is  not  desirable  to  give  him  the  power  output  of  every 
component  of  the  system. 
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QUESTION  DATA  SHEET 


Question  Number  40 


QUESTION:  The  system  software  may  be  reloaded  quickly. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Some  computerized  systems  require  the  operator  to  load  new  or 
different  software  at  regular  or  frequent  intervals.  The  quicker  this  can  be 
done,  the  less  demanding  and  frustrating  it  will  be  to  the  operator.  For  those 
systems  where  the  mission  software  is  reloaded  by  someone  other  than  the  prime 
mission  operator,  such  reloading  still  has  an  impact  on  the  prime  mission 
operator  in  areas  such  as  idle  time  and  workload  backlog. 

EXAMPLES:  The  operator  is  able  to  load  the  software  from  a  disk  by  merely 
setting  some  toggle  switches  and  depressing  a  "Reboot"  button. 

GLOSSARY :  Reloaded:  reboot,  reprogram. 


AFTECP  800-2,  Vol  4  Attachment  1  1  Nov  82 


59 


QUESTION  DATA  SHEET 


Question  Number  41 


QUESTION:  The  system  software  needs  to  be  reloaded  infrequently. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  An  aircrew  training  device  is  usually  reloaded  quickly  and  easily 
with  little  downtime;  however,  training  is  interrupted  and  realism  is  destroyed. 
Therefore,  reload  is  considered  a  reason  for  a  low  rating. 
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QUESTION  DATA  SHEET 


Question  Number  42 


QUESTION:  System  warm-up  time  is  small. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  A  machine  which  requires  large  amounts  of  warm-up  time  before  use 
causes  the  operator  to  waste  time,  lowers  productivity,  and  often  frustrates  the 
operator.  For  systems  which  run  continuously,  this  question  only  applies  to 
power  failures,  changing  software  loads,  bringing  peripherals  on  line,  instru¬ 
ment  warm-up,  etc. 
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QUESTION  DATA  SHEET 


Question  Number  43 


QUESTION:  The  operator's  manual  makes  minimal  use  of  cross-references. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Excessive  cross-referencing  in  the  operators  manual  forces  extra 
work  on  the  operator.  The  worst  case  happens  when  the  operator  has  used  all 
fingers  and  toes  and  forgotten  the  original  question.  Cross-referencing  should 
be  restricted  to  cases  where  it  serves  a  useful  purpose. 
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QUESTION  DATA  SHEET 


Question  Number  44 


QUESTION:  It  is  easy  to  locate  specific  information  within  the  operator's 
manual . 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Manuals  which  an  operator  must  or  might  use  during  operation  of  a 
machine  should  be  arranged,  indexed,  and  have  a  table  of  contents  such  that  the 
operator  can  locate  specific  information  rapidly. 

EXAMPLES:  The  evaluator  should  think  up  some  information  desirable  to  retrieve 
from  the  operator's  manual,  then  see  how  hard  it  is  to  find  it.  This  process 
may  be  repeated  several  times. 
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QUESTION  DATA  SHEET 

Question  Number  45 

QUESTION:  The  operator's  manual  is  a  reasonable  size. 

CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  Oversize  operator  manuals  are  physically  difficult  to  work  with 
as  well  as  reflecting  poor  organization  and  lack  of  consideration  of  what  the 
operator  needs  to  know. 
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QUESTION  DATA  SHEET 


Question  Number  46 


QUESTION:  The  operator  performs  no  tedious  functions  which  could  be  handled  by 
the  system. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Functions  which  could  be  performed  by  the  machine  but  which  are 
forced  upon  the  operator  represent  a  source  of  additional  workload  for  the 
operator. 

EXAMPLES:  1.  Hand  copying  of  information  from  computer  display  to  records. 

2.  Entry  of  degree  signs  and  decimal  points. 

3.  Repeated  entry  of  duplicated  data,  such  as  the  words  "Year"  and 
"Month"  when  entering  dates. 
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QUESTION  DATA  SHEET 


Question  Number  47 


QUESTION:  The  operator  is  rarely  bored  and  performs  a  dynamic  function. 
CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  A  well-designed  system  takes  into  consideration  the  best  cap¬ 
abilities  of  the  computer  (repetitive  tasks)  and  the  best  capabilities  of  the 
operator  (decision  making,  experience). 

EXAMPLES:  The  machine  should  allow  the  operator  to  make  decisions  which  save 
time  and  increase  productivity,  such  as  skipping  tests  which  the  operator 
already  knows  the  "black  box"  will  pass. 
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QUESTION  DATA  SHEET 


Question  Number  48 


QUESTION:  The  operator  is  not  forced  to  wait  for  the  machine  to  respond. 
CHARACTERISTIC:  Workload  reasonabi 1 ity. 

EXPLANATIONS:  An  operator  who  is  continuously  forced  to  wait  for  the  machine  to 
become  available  for  use  is  less  than  optimally  productive  and  is  probably 
frustrated.  Machines  which  respond  slowly  are  probably  overloaded.  Consider 
the  action  taken  and  the  response  time  of  the  system  as  it  relates  to  mission 
accompl i shment. 
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QUESTION  DATA  SHEET 


Question  Number  49 


QUESTION:  The  operator  is  not  a  slave  to  the  machine. 

CHARACTERISTIC:  Workload  reasonability. 

EXPLANATIONS:  Operators  which  are  "chained"  to  a  machine  get  worn  out  rapidly, 
tend  to  become  inattentive,  and  make  more  and  more  errors  as  time  passes. 
Furthermore,  they  have  little  job  satisfaction  and  have  a  high  turn-over  rate, 
resulting  in  increased  training  costs  and  low  organizational  productivity. 

EXAMPLES:  Automobile  assembly-line  workers  are  perfect  examples  of  operators 
‘'chained"  to  a  machine. 
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WORKLOAD  REASONABILITY.  A  computerized  system  contains  the  quality  of  workload 
reasonability  to  the  extent  that  the  tasks  required  of  the  operator  are  within 
the  operator's  capabilities  and  the  extent  to  which  the  operator  performs  a 
useful,  meaningful  role. 

An  optimum  design  of  a  system  which  involves  an  operator  and  a  computerized 
machine  takes  advantage  of  the  best  capabilities  of  both:  the  machine  to 
perform  repetitive  tasks  rapidly  and  the  operator  to  make  command  decisions 
involving  unusual  situations. 
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D.  DESCRIPTIVENESS.  A  computerized  system  contains  the  quality  of  descrip 
tiveness  to  the  extent  that  the  operator  has  available  adequate  explanations  of 
every  function  the  operator  is  required  to  perform  and  every  function  the 
machine  performs. 

The  operator  need  not  be  informed  in  detail  of  every  task  the  machine 
performs,  but  there  definitely  are  certain  things  the  operator  must  know  to 
fulfill  the  mission.  The  questionnaire  relies  upon  the  knowledge  of  the  operator 
to  define  what  it  is  the  operator  needs  to  know. 
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QUESTION  DATA  SHEET 


Question  Number  50 


QUESTION:  Power-on  and  power-off  procedures  are  well  documented. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Since  the  machine  cannot  aid  the  operator  during  power-on  and 
power-off  activities,  an  easy-to-use,  understandable,  and  correct  set  of  written 
procedures  must  be  available  to  the  operator. 


^>>> 
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QUESTION  DATA  SHEET 


Question  Number  51 


QUESTION:  The  operator  has  adequate  Instructions  for  handling  emergencies. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  The  operator  must  have  adequate  emergency  instructions  available 
if  he  is  to  avoid  harm  to  personnel  or  damage  to  equipment.  This  includes 
emergency  power-on  and  power-off  Instructions. 
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QUESTION  DATA  SHEET 


Question  Number  52 


QUESTION:  Legitimate  responses  for  all  conditions  are  explained. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  In  order  for  a  system  to  be  considered  properly  described,  al  1 
situations  which  the  normal  operator  will  encounter  must  be  addressed  by  either 
the  documentation  or  by  computer-generated  descriptions. 

GLOSSARY :  Explained:  documented  and/or  prompted  by  the  software. 


74 


AFTECP  800-2,  Vol  4  Attachment  1  1  Nov  82 


QUESTION  DATA  SHEET 

Question  Number  53 


QUESTION:  The  software  provides  a  question-answer  type  aid. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  The  availability  of  question-answer  operator  aids  can  greatly  aid 
operator  work  by  making  technical  information  quickly  and  readily  available. 

EXAMPLES: 

1.  One  system  uses  a  "help  me"  command.  If  the  operator  needs  information 
on  how  to  format  a  tape  advance  command,  for  instance,  he  enters  "HELP  ME,  TAPE 
ADVANCE"  and  he  is  instantly  presented  with  a  detailed  explanation  of  various 
tape  control  commands. 

2.  Another  system  responds  with  a  more  detailed  data  request  whenever  the 
operator  enters  a  question  mark.  For  Instance  the  operator  leaves  the  machine 
and  later  returns  to  find  the  machine  requesting  "ENTER  ALTITUDE."  The  operator 
has  forgotten  which  altitude  was  being  requested  when  he  left.  A  question  mark 
is  entered  and  the  machine  responds  "ENTER  SIMULATED  AIRCRAFT  ALTITUDE  IN 
THOUSANDS  OF  FEET." 
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QUESTION  DATA  SHEET 

Question  Number  54 


QUESTION:  The  system  will  explain  each  command  upon  user's  request. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Descriptions  of  operator  commands  should  be  readily  available  to 
the  operator.  This  is  especially  helpful  when  the  operator  makes  regular  use  of 
a  very  large  number  of  commands. 

EXAMPLES:  One  system  automatically  displays  a  short  explanation  of  what  is 
required  anytime  an  illegal  command  is  entered.  Therefore,  anytime  the  operator 
needs  more  information  before  making  his  decision(s),  he  enters  an  "X"  or  other 
illegal  command.  The  machine  then  gives  him  a  short  summary  of  legal  responses 
to  the  situation  and  automatically  re-asks  the  original  question. 
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QUESTION  DATA  SHEET 

Question  Number  55 

QUESTION:  Explanations  of  how  to  interpret  all  output  data  are  available. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS: 

EXAMPLES:  Explanations  may  be  available  through  an  on-line  question-type  aid  or 
through  the  operator's  manual. 
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QUESTION  DATA  SHEET 


Question  Number  56 


QUESTION:  The  operator  is  adequately  alerted  when  the  system  requires  operator 
action. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  When  the  machine  is  performing  a  long  task,  which  requires  no 
operator  actions,  the  operator  may  do  something  else,  such  as  take  a  coffee 
break  or  carry  on  a  conversation.  However,  if  no  time  is  to  be  wasted,  he  must 
be  told  when  to  end  the  break. 

EXAMPLES:  Alerted: 

Optimum:  a  combination  of  audible  and  visual  signals. 

Acceptable:  either  audible  or  visual  prompts. 

Unacceptable:  no  signal  other  than  apparent  machine  inactivity. 
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QUESTION  DATA  SHEET 

Question  Number  57 


QUESTION:  The  machine  gives  the  operator  decision  aids  if  tasks  cannot  be 
executed  as  ordered. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  When  a  task  cannot  be  performed  as  ordered  by  the  operator,  the 
machine  should  display  information  to  aid  the  operator  in  making  a  decision 
about  alternative  ways  to  get  the  job  done. 

EXAMPLES: 

1.  If  an  automatic  test  equipment  operator  has  a  test  stop  because  a 
certain  power  supply  is  not  installed  in  the  test  equipment,  then  the  machine 
should  list  other  power  supplies  available  for  replacements.  The  operator  would 
then  be  able  to  choose  a  different  power  supply  and  continue  the  test. 

2.  On  a  simulator,  the  machine  should  tell  the  operator  exactly  what 
procedures  to  use  to  simulate  a  partial  mission  if  one  of  the  student  stations 
is  inoperative. 
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QUESTION  DATA  SHEET 


Question  Number  58 


QUESTION:  The  version  number  (revision  number)  of  the  software  is  readily 
available  to  the  operator. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS :  Frequent  software  updates  dictate  that  the  operator  be  aware  of 
exactly  which  version  of  an  operational  program  is  being  used. 

EXAMPLES: 

1.  An  automatic  test  equipment  operator  must  usually  use  a  new  test 
program  after  a  Time  Compliance  Technical  Order  (TCTO)  has  been  implemented  on  a 
given  black  box.  Modified  black  boxes  will  often  fail  an  old  test  program  even 
though  they  are  good. 

2.  In  Aircrew  Training  Devices,  the  version  number  is  normally  carried  on 
a  Form  781. 


80 


AFTECP  800-2,  Vol  4  Attachment  1  1  Nov  82 


QUESTION  DATA  SHEET 


Question  Number  59 


QUESTION:  Data  base  configuration  data  are  readily  available  to  the  operator. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Many  systems  use  data  bases,  the  configuration  of  which  is  as 
important  as  the  configuration  of  the  computer  programs. 

EXAMPLES: 

1.  Simulators:  The  flight  characteristics  of  a  given  aircraft  may  exist 
as  a  data  base. 

2.  Automatic  Test  Equipment:  Parameters  of  stimulus  and  measurement 

devices  may  exist  as  a  data  base. 

3.  Mission  Data  Preparation:  Mission  data  tapes  of  latitudes  and  longi¬ 
tudes  from  a  data  base  containing  thousands  or  millions  of  mensurated  points. 

GLOSSARY:  Data  base:  a  collection  of  numeric  parameters  which  describe  an 
entity  such  as  flight  characteristics  of  an  aircraft  or  the  contents  of  all 
accounts  with  a  credit  union. 

Configuration  identification:  some  means  of  knowing  which  data  base 
is  which  (usually  an  alpha-numeric  system). 
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QUESTION  DATA  SHEET 


Question  Number  61 


QUESTION:  The  operator's  manual  clearly  explains  the  normal  sequential  steps  of 
operation . 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  Lack  of  descriptions  of  normal  sequential  steps  of  operation 
results  in  poor  training  and  in  operator  uncertainty.  Poorly  organized  manuals 
make  the  operator  hunt  and  dig  through  all  the  information  available  to  figure 
out  what  may  be  simple,  straight-forward  "normal"  procedures.  The  normal 
sequential  steps  of  operations  should  not  be  hidden  amid  verbiage  describing 
unusual  situations. 
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QUESTION  DATA  SHEET 


Question  Number  62 


QUESTION:  The  operator's  manual  contains  a  useful  table  of  contents. 
CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  A  table  of  contents  is  the  key  to  the  organization  and  content  of 
an  operator1 s  manual . 

GLOSSARY :  Useful :  The  table  of  contents  not  only  exists,  but  actually  reflects 
the  organization  and  content  of  the  operator's  manual. 
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QUESTION  DATA  SHEET 


Question  Number  63 


QUESTION:  The  operator's  manual  contains  a  useful  index. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  An  index  is  required  to  allow  the  operator  to  locate  specific 
information  within  the  operator's  manual  by  subject  matter.  Important  subject 
matter  often  does  not  appear  in  the  table  of  contents  under  a  heading  the 
operator  is  familiar  with. 

EXAMPLES:  The  table  of  contents  may  have  a  300-page  section  titled  "Malfunc¬ 
tions,"  whereas  the  operator  would  find  "engine  bleed  air  malfunctions"  through 
the  index  without  leafing  through  the  300  pages. 

GLOSSARY :  Index:  an  alphabetical  listing  of  names  and  topics  and  the 

corresponding  page  numbers  on  which  they  are  discussed. 
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QUESTION  DATA  SHEET 


Question  Number  64 


QUESTION:  The  operator's  manual  contains  a  useful  glossary. 

CHARACTERISTIC:  Descriptiveness. 

EXPLANATIONS:  A  glossary  of  terminology  is  often  essential  to  an  operator's 
manual  which  is  clear,  concise,  and,  above  all,  descriptive. 

GLOSSARY :  Glossary:  a  list  of  terms  used  in  the  manual  and  their  corresponding 
definitions. 
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DESCRIPTIVENESS.  A  computerized  system  contains  the  quality  of  descriptiveness 
to  the  extent  that  the  operator  has  available  adequate  explanations  of  every 
function  the  operator  is  required  to  perform  and  every  function  the  machine 
performs.  The  operator  need  not  be  informed  in  detail  of  every  task  the  machine 
performs,  but  there  definitely  are  certain  things  the  operator  must  know  to 
fulfill  the  mission.  The  questionnaire  relies  upon  the  knowledge  of  the  operator 
to  define  what  it  is  the  operator  needs  to  know. 
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E.  CONSISTENCY.  A  computerized  system  contains  the  quality  of  consistency  to 
the  extent  that  the  behavior  of  the  machine  and  documentation  corresponds  to  the 
expectations  of  the  operator. 

There  should  be  a  near  one-to-one  correspondence  between  what  the  machine 
does,  what  the  documentation  says  it  will  do,  and  what  the  operator  has  been 
trained  to  expect  the  machine  to  do. 
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QUESTION  DATA  SHEET 


Question  Number  65 


QUESTION:  The  operator  uses  systematic  formats  to  enter  commands. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  To  operate  the  machine,  the  operator  must  memorize  a  set  of 
command  instructions.  The  more  systematic  the  naming  convention  and  format  used 
for  the  commands,  the  easier  it  is  to  memorize  and  use. 

GLOSSARY :  Systematically  formatted:  It  should  be  obvious  that  the  software 
designers  used  a  naming  convention  and  formatting  convention  in  designing  the 
set  of  commands  which  an  operator  must  enter. 
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QUESTION  DATA  SHEET 


Question  Number  66 


QUESTION:  The  commands  are  entered  in  a  standard  manner. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  Use  of  a  standard  entering  technique  has  several  advantages: 

1.  The  new  operator  may  already  know  it. 

2.  Manuals  describing  it  are  available. 

3.  Usually  more  people  are  around  who  can  offer  assistance. 

EXAMPLES:  If  a  keyboard  is  used,  it.  should  be  a  standard  typewriter  keyboard. 
Likewise,  numeric  key  pads,  curser  arrow  key  pads,  and  light  pens  should  be 
standard. 
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QUESTION  DATA  SHEET 


Question  Number  67 


QUESTION:  Requirements  for  operator  input  agree  with  the  operator's  manual. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  When  there  is  inconsistency  between  the  operator's  manual  and 
actual  operations,  it  leads  to  operator  confusion,  frustration,  errors,  and 
reaccompl i shment. 
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QUESTION  DATA  SHEET 


Question  Number  68 


QUESTION:  Messages  to  the  operator  are  systematically  formatted. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS :  When  messages  to  the  operator  are  systematically  formatted  they 
aid  the  operator  in  that  he  can  read  faster  and  always  knows  where  to  look 
within  a  message  to  get  exactly  the  information  desired. 

GLOSSARY :  Systematically  Formatted:  It  should  be  obvious  that  the  software 
designers  used  both  a  naming  convention  and  a  formatting  convention. 
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QUESTION  DATA  SHEET 


Question  Number  69 


QUESTION:  Messages  requiring  action  by  the  operator  are  always  highlighted  in 
some  fashion. 


CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  To  minimize  the  necessity  for  the  operator  to  constantly  watch 
the  machine,  the  machine  must  tell  the  operator  every  time  it  needs  operator 
attention,  and  it  should  always  do  it  in  the  same  manner  so  that  the  operator 
knows  what  to  expect. 

EXAMPLES:  A  combination  of  audio  and  visual  signals  is  best.  Either  audio  or 
visual  is  next  best;  in  any  case,  the  operator  should  always  be  alerted  in  the 
same  manner. 
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QUESTION  DATA  SHEET 

Question  Number  70 


QUESTION:  Operator  entries  always  result  in  some  type  of  response. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  For  the  operator  to  know  that  the  machine  has  deciphered  the 
operator  commands,  and  is  taking  action  upon  them,  the  machine  must  so  Indicate 
to  the  operator. 

EXAMPLES:  One  system  responds  with  an  asterisk,  another  simply  moves  the 
display  cursor.  Yet  another  system  replies  with  ''COMMAND1'  each  time  it  has 
completed  action  on  the  previous  command. 
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QUESTION  DATA  SHEET 


Question  Number  71 


QUESTION:  Response  times  are  similar  for  groups  of  similar  activities. 
CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  To  minimize  operator  frustration,  the  machine  must  "behave" 
similarly  in  similar  situations.  Response  time  is  an  important  factor  in 
measuring  similar  behaviors. 

Another  system  which  would  minimize  operator  frustration  concerning  re¬ 
sponse  time  would  be  for  the  system  to  display  to  the  operator  the  amount  of 
time  remaining  before  the  system  would  require  operator  action. 

EXAMPLES:  Remote  terminal  systems  have  notoriously  Inconsistent  -esponse  times. 
Some  systems  have  response  times  that  can  vary  between  .1  seco:.—  and  20  min¬ 
utes. 
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QUESTION  DATA  SHEET 


Question  Number  72 


QUESTION:  System  performance  corresponds  with  documented  performance  (speci¬ 
fications,  user  manuals,  etc.). 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  Much  instruction  and  almost  all  troubleshooting  depends  heavily 
upon  documentation.  If  machine  performance  does  not  agre$  with  what  is  docu¬ 
mented,  then  the  value  of  formal  training  is  decreased,  and  troubleshooting 
efforts  (as  well  as  dally  operations)  are  made  more  difficult. 
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QUESTION  DATA  SHEET 


Question  Number  73 


QUESTION;  Checklists  agree  with  the  operator's  manual. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  A  checklist  should  be  a  summary  of  essential  Information  con¬ 
tained  in  the  operator's  manual.  It  should  not  be  developed  independently  of 
the  operator's  manual.  If  regulations  prohibit  using  checklists,  the  operator 
should  answer  as  to  whether  they  would  be  beneficial  (reference  section  B, 
paragraph  7.e). 
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QUESTION  DATA  SHEET 


Question  Number  74 


QUESTION:  Operator  manuals  are  systematically  formatted. 

CHARACTERISTIC:  Consistency. 

EXPLANATIONS:  In  cases  where  there  Is  more  than  one  operator's  manual,  all  such 
manuals  should  be  laid  out  the  same  way.  Thus  operators  who  must  use  more  than 
one  such  manual  may  pay  attention  to  content  rather  than  wasting  time  trying  to 
locate  information. 

EXAMPLES:  There  Isa  standard  for  the  layout  of  aircraft  -1  manuals. 
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CONSISTENCY.  A  computerized  system  contains  the  quality  of  consistency  to  the 
extent  that  the  behavior  of  the  machine  and  documentation  corresponds  to  the 
expectations  of  the  operator.  There  should  be  a  near  one-to-one  correspondence 
between  what  the  machine  does,  what  the  documentation  says  It  will  do,  and  what 
the  operator  has  been  trained  to  expect  the  machine  to  do. 
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F.  SIMPLICITY.  A  computerized  system  contains  the  quality  of  simplicity  to  the 
extent  that  information  presented  to  the  operator  or  entered  by  the  operator  is 
grouped  into  short,  readily  understandable  structures. 

Complicated  data  structures,  data  entry  formats,  or  operator  manuals  all 
require  the  operator  to  spend  more  time  in  developing  an  understanding  of  the 
system,  and  may  have  a  tendency  to  confuse  the  operator  as  well. 
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QUESTION  DATA  SHEET 

Question  Number  75 


QUESTION:  If  different  operator  stations  doing  similar  functions  exist,  the 
input  commands  on  the  various  stations  are  similar. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  The  less  the  operator  needs  to  know,  the  easier  it  is  to  know  it 
well .  If  he  must  know  two  commands  to  operate  the  system,  the  task  of  learning 
the  operating  instructions  Is  about  twice  as  complex. 

EXAMPLES:  Automatic  Test  Equipment:  commands  used  to  test  various  line-re¬ 
placeable  units  should  be  the  same  where  the  functions  of  the  command  is  the 
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QUESTION  DATA  SHEET 

Question  Number  76 

QUESTION ;  Operator  entered  Instructions  are  relatively  short. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  In  general,  the  shorter  an  instruction  the  easier  It  Is  to 
formulate  because  It  contains  fewer  Ideas  which  must  be  combined  to  build  the 
Instruction. 
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QUESTION  DATA  SHEET 


Question  Number  77 


QUESTION:  It  is  easy  to  understand  actions  required  of  the  operator. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  In  order  to  make  it  easy  for  the  operator  to  understand  what  is 
supposed  to  be  accomplished,  the  instructions  must  be  relatively  simple  and  easy 
to  understand. 
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QUESTION  DATA  SHEET 


Question  Number  78 


QUESTION:  Messages  to  the  operator  are  short. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Shorter  messages  generally  contain  less  information  than  long 
messages.  The  less  information  contained  in  a  message,  the  easier  it  is  under¬ 
stood.  This  includes  messages  on  console  displays  and  printouts. 
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QUESTION  DATA  SHEET 

Question  Number  79 

QUESTION:  Each  new  message  contains  only  one  idea  to  which  the  operator  must 
respond. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  The  simplest  message  is  one  that  contains  only  one  idea.  The 
simpler  a  message,  the  easier  it  is  to  understand. 
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QUESTION  DATA  SHEET 

Question  Number  80 


QUESTION:  Only  essential  or  useful  information  is  displayed  to  the  operator. 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Excess  information  makes  it  more  difficult  to  find  required 
information  and  the  nonessential  information  may  confuse  the  operator.  The 
amount  of  data  displayed  to  the  operator  should  not  be  a  hindrance  to  under¬ 
standing. 

EXAMPLES:  Displays  can  be  expanded  or  decreased  so  that  a  240  by  240  mile  area 
can  be  decreased  to  a  60  by  60  mile  area  of  concern. 
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QUESTION  DATA  SHEET 

Question  Number  81 

QUESTION:  The  display  is  not  overcrowded  (unless  commanded  to  be  so). 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Crowded  displays  decrease  the  ability  to  organize  the  Information 
displayed.  The  less  organized  a  display,  of  course,  the  harder  it  is  to  find 
useful  information  therein. 
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QUESTION  DATA  SHEET 

Question  Number  82 

QUESTION:  Difficult  words  or  characters  are  rarely  used. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Using  a  hard-to-understand  word  or  symbol  in  a  message  makes  the 
whole  message  hard  to  understand. 


QUESTION:  Data  formats  are  easily  understandable. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  For  an  operator  to  be  able  to  rapidly  withdraw  necessary  data 
from  a  data  structure,  the  structure  of  the  data  must  enhance  understanding. 

GLOSSARY :  Data  structure:  a  list  or  table  of  data,  such  as  a  malfunction  list 
or  parameter  list. 
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QUESTION  DATA  SHEET 


Question  Number  84 


QUESTION:  The  operator  has  appropriate  checklists  available. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Excess  information  makes  understanding  of  essential  information 
difficult.  For  an  experienced  operator,  checklists  should  contain  essential 
Information  condensed  from  the  detailed  procedures  manuals. 
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QUESTION  DATA  SHEET 

Question  Number  85 


QUESTION:  The  number  of  checklists  required  Is  manageable. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  A  single  checklist  Is  the  ideal  situation  for  a  given  operator's 
station.  No  operator  should  be  required  to  use  more  than  two  or  three  check¬ 
lists  simultaneously. 
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QUESTION  DATA  SHEET 

Question  Number  86 

QUESTION:  The  operator's  manual  Is  a  single  volume  (except  for  checklists). 
CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Multi  volume  operator  manuals  represent  a  level  of  complexity 
higher  than  single-volume  operator  manuals.  Classified  second  volumes  are 
necessary,  but  still  more  complex  than  single  volumes. 
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QUESTION  DATA  SHEET 


Question  Number  87 


QUESTION:  The  operator's  manual  is  easy  to  understand. 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  In  general,  the  easier  an  operator's  manual  is  to  understand,  the 
easier  it  Is  to  use,  the  more  it  will  be  used,  and  the  more  it  will  be  used 
correctly. 
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QUESTION  DATA  SHEET 

Question  Number  88 


?UESTI0N:  Alternatives  to  normal  operating  sequences  are  described  separately 
not  embedded  within  normal  procedures). 

CHARACTERISTIC:  Simplicity. 

EXPLANATIONS:  Excess  information  makes  understanding  of  essential  information 
difficult.  Alternative  procedures  usually  represent  excess  information  if 
contained  within  the  normal  procedures  (which  hopefully  are  most  often  used). 
Alternative  procedures  should,  therefore,  be  contained  in  a  separate  section  of 
the  documentation.  .  „ 
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SIMPLICITY.  A  computerized  system  contains  the  quality  of  simplicity  to  the 
extent  that  information  presented  to  the  operator  or  entered  by  the  operator  is 
grouped  Into  short,  readily  understandable  structures. 

Complicated  data  structures,  data  entry  formats,  or  operator  manuals  all 
require  the  operator  to  spend  more  time  in  developing  an  understanding  of  the 
system,  and  may  have  a  tendency  to  confuse  the  operator  as  well. 
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G.  GENERAL  QUESTIONS:  The  following  section  contains  general  questions  about 
characteristics  of  the  system.  The  evaluator  should  formulate  a  subjective 
opinion  based  on  general  feelings  without  reference  to  previous  answers. 
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QUESTION  DATA  SHEET 


Question  Number  89 


QUESTION:  The  concepts  of  assurability  as  implemented  in  the  system  contribute 
to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Assurability:  A  computerized  system  contains  the  quality  of  assur¬ 
ability  to  the  extent  that  it  aids  the  operator  In  validating  data,  avoiding 
errors,  and  correcting  errors  once  made. 

A  system  which  has  been  designed  to  aid  the  operator  in  error  avoidance  may 
or  may  not  have  good  assurability.  A  system  should  also  be  designed  so  that 
errors  are  easy  to  correct,  and,  above  all,  so  that  errors  do  not  have  catas¬ 
trophic  effects. 
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QUESTION  DATA  SHEET 


Question  Number  90 


QUESTION:  The  concepts  of  controllability,  as  implemented  in  the  system, 

contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Control labi 1 ity:  A  computerized  system  contains  the  quality  of 
controllability  to  th-a  extent  that  it  allows  the  operator  to  direct  the  opera¬ 
tions  of  the  machine. 

The  operator  must  be  able  to  direct  or  control  the  operation  of  the  machine 
in  order  to  utilize  it  effectively  and  efficiently. 
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QUESTION  DATA  SHEET 

Question 


QUESTION:  The  concepts  of  workload  reasonability,  as  implemented  in  th 
contribute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Workload  reasonability.  A  computerized  system  contains  th 
of  workload  reasonability  to  the  extent  that  the  tasks  required  of  the 
are  within  the  operator's  capabilities  and  the  extent  to  which  the 
performs  a  useful,  meaningful  role. 

Optimum  design  of  a  system  which  involves  an  operator  and  a  corn 
machine  takes  advantage  of  the  best  capabilities  of  both:  the  m: 
perform  repetitive  tasks  rapidly  and  the  operator  to  make  command 
Involving  unusual  situations. 
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QUESTION  DATA  SHEET 

Question  Number  92 

QUESTION:  The  concepts  of  descriptiveness  as  implemented  in  the  system  contrib¬ 
ute  to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Descriptiveness:  A  computerized  system  contains  the  quality  of 
descriptiveness  to  the  extent  that  the  operator  has  available  adequate  explana¬ 
tions  of  every  function  the  operator  is  required  to  perform,  and  every  function 
the  machine  performs. 

The  operator  need  not  be  informed  in  detail  of  every  task  the  machine 
performs,  but  there  definitely  are  certain  things  the  operator  must  know  to 
fulfill  the  mission.  The  questionnaire  relies  upon  the  knowledge  of  the  operator 
to  define  what  it  is  the  operator  needs  to  know. 
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QUESTION  DATA  SHEET 


Question  Number  33 


QUESTION:  The  concepts  of  consistency  as  implemented  in  the  system  contribute 
to  the  usability  of  the  system. 

CHARACTERISTIC:  General  questions. 

EXPLANATIONS: 

GLOSSARY :  Consistency:  A  computerized  system  contains  the  quality  of  consis¬ 
tency  to  the  extent  that  the  behavior  of  the  machine  and  documentation  cor¬ 
responds  to  the  expectations  of  the  operator. 

There  should  be  a  near  one-to-one  correspondence  between  what  the  machine 
does,  what  the  documentation  says  It  will  do,  and  what  the  operator  has  been 
trained  to  expect  the  machine  to  do.  Furthermore,  the  documentation  normally 
available  to  the  operator  should  agree. 
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QUESTION  DATA  SHEET 

Question  Number  94 

QUESTION:  The  concepts  of  simplicity  as  implemented  in  the  system  contribute  to 
usability  of  the  system. 

CHARACTERISTIC:  General  questions. 


K 


EXPLANATIONS: 

GLOSSARY :  Simplicity:  A  computerized  system  contains  the  quality  of  simplicity 
to  the  extent  that  information  presented  to  the  operator  or  entered  by  the 
operator  is  grouped  into  short,  readily  understandable  structures. 

Complicated  data  structures,  data  entry  formats,  or  operator's  manuals  all 
require  the  operator  to  spend  more  time  in  developing  an  understanding  of  the 
system,  and  may  have  a  tendency  to  confuse  the  operator  as  well. 
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QUESTION  DATA  SHEET 


Question  Number  95 


QUESTION:  Overall  It  appears  that  the  operator-machine  Interface  has  been 
well  designed. 

CHARACTERISTIC:  General  questions 

EXPLANATIONS:  This  Is  the  bottom-line,  most  highly  subjective  question  asked. 
Rate  the  system  according  to  your  personal  feeling. 
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ATTACHMENT  2 

Subject  Matter  Versus  Question  Number  Cross-Reference  (Index) 

This  attachment  contains  Information  that  should  help  the  evaluator  locate 
specific  question  subject  matter  and  glossary  definitions.  The  Index  contains 
question  subjects  listed  alphabetically  and  cross-referenced  to  the  question 
addressing  that  subject. 
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Normal  35,61,88 
Operator 

Action  Required  4,56,69,77 
Aid  34,53,57 
Alert  15,56 
Commands  65,75,76 
Functions  46-49 
Input  1-6,67,70 
Messages  37-39,68,69,78 
Must  Respond  69,  79 

Operators  Manual  35,43-45,61-64,67,72-74,84-88 
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Device  Selection  20 
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Messages  37-39,68 
Quantity  29 
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Overcrowded  81 
Partial  Failure  30 
Performance  72 
Pointer  8 
Power-off (on)  50 
Preparation  32 
Procedures  50 
Program  (Software)  40,41 
Quantity  29,36 
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Rarely  47 
Rate  (Speed)  38 
Recording  Data  11 
Reference  35,43,60 
Reload  Software  40 
Request  54 
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Time  48,71 
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Device  Off-Line  13 
Mode  of  Operation  28 
Self-Type  19 
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Self-Test  19 
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Sequential  61 
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Simplicity  75-88,94 
Size  45 
Slave  49 
Small  36,39,43 
Software  40,41,53,58 
Specifications  72 
Standardized  Language  66 
Status  21,23 
Strings  26 

Symbol  (Character)  10,82 
System 

Configuration  23 
Failure  1,30 
Halt  16 

Needs  Operator  56 
Operation  35,36 
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Warm-up  42 

Systematic  65,68,74 

Table  of  Contents  62 

Task 

Abort  12 

Cannot  Be  Done  57 

Data  31 

Interrupt  12 

Tedious  46 

Time  42,71 

Understand  37,77,83 

Validity  7,8 

Version  Number  58,59 

1 

.N 

:> 

fa 

3 

Visibility  27 

Walt  48 

Warm-up  42 

Word  82 

Workload  Reasonability  31-49,91 

